RELEASE VERSION = 2.4.0.72
Intel® QuickAssist Technology (Intel® QAT) Driver for VMware ESXi*
===============================================================================
This software enables Intel® QuickAssist Technology (Intel® QAT) acceleration
on VMware ESXi.
It is using SR-IOV technology to allow sharing of single Physical
Function (PF) resources - Virtual Functions (VFs) - and provide acceleration in
multiple Virtual Machines (VMs).

This document describes how to install Intel® QAT Host driver on VMware ESXi
hypervisor and how to make Intel® QAT HW available inside guests.
For more details on how to use Intel® QAT from a Virtual Machine,
please refer to the documents mentioned in the "Documentation" section.

Note that this document assumes that the reader is familiar with virtualization
technologies and has some level of familiarity with the VMware ESXi hypervisor.
This document does not explain how to install VMware ESXi, how to install
a virtual machine and how to administrate ESXi using vSphere* Host Client or
VMware vCenter*.
For more details on these topics please refer to VMware's documentation.

*Other names and brands may be claimed as the property of others.


License
=======
Refer to LICENSE.txt in this package for license information before using
this software.


Details/Limitations of this Release
===================================
* This software release is intended for platforms that contain:
  * 4th Generation Intel® Xeon® Scalable Processor (4xxx QAT)
* ESXi limitation: number of PCI passthrough devices per VM is limited, so
  please check KB article (https://kb.vmware.com/s/article/1003497) for exact
  limits. ESXi will not allow to power on VM if such limit is exceeded.


Documentation
=============
Associated software and collateral can be found on the open source website:
https://developer.intel.com/quickassist

Additional documentation includes:
 * Intel® QuickAssist Technology Software for Linux* - Release Notes - Hardware
   Version 2.0
 * Intel® QuickAssist Technology Software for Linux* - Getting Started Guide -
   Hardware Version 2.0
 * Intel® QuickAssist Technology Software for Linux* - Programmer's Guide -
   Hardware Version 2.0
 * Intel® QuickAssist Technology Software for Windows* - Release Notes
 * Intel® QuickAssist Technology Software for Windows* - Technical Guide
 * Intel® QuickAssist Technology API Programmer's Guide
 * Intel® QuickAssist Technology Cryptographic API Reference Manual
 * Intel® QuickAssist Technology Compression API Reference Manual


Installing Intel® QAT VMware Driver
===================================
1) Open a Secure Shell (SSH) connection to the target ESXi host (SSH needs to
   be enabled to perform this operation).
2) Copy the component bundle to the ESXi server.
   Technically, you can place the file anywhere that is accessible to the ESXi
   console shell, but for these instructions, we'll assume the location
   is in '/tmp' folder.
   Here's an example of using the Linux `scp` utility to copy the file from
   downloaded location to the remote/target ESXi server located at 10.10.10.10:
    > scp qat-2.0_*.tar.gz root@10.10.10.10:/tmp
3) Extract the package (assuming it has been copied to /tmp folder)
    > cd /tmp
    > tar -xzf qat-2.0_*.tar.gz
4) There are two ways to install the driver:
    a) From component via the following command (recommended approach):
        > esxcli software component apply \
          -d /tmp/*-qat-2.*.zip
    b) From the VIB package extracted from component:
        > unzip /tmp/*-qat-2.*.zip
        > esxcli software vib install \
          -v /tmp/vib20/qat/INT_bootbank_qat_2.*.vib

   Note: This step can take some time (30+ seconds).
5) Reboot the system to complete driver installation:
    > reboot
6) If the Intel® QAT driver has been loaded without errors, you should see
   the qat module showing in the list of system modules:
    > esxcfg-module -l | grep qat
    > qat               8    3968


Using SR-IOV VFs
================

Enabling SR-IOV
---------------
1) Login to the target ESXi host via vSphere Host Client.
2) In the left pane, expand "Host" and click on "Manage" section.
3) In the center pane, choose "Hardware" tab and "PCI Devices" sub-tab.
4) Find devices with "Intel Corporation QAT" text in "Description"
   and click on "Configure SR-IOV" button.
5) In dialog window change "Enabled" param to "Yes" and enter the
   desired number of VF in the range between 1 and maximum indicated in window.
6) Click "Save" and ESXi will reconfigure device with requested number of VFs.
   On this step, ESXi may notify you that SR-IOV configuration changes may
   not take effect until the host is restarted. If you don't see VFs
   listed in UI, please follow VMware recommendation and reboot system.
7) Now the Intel® QAT VFs are enabled in the system. Intel® QAT VFs should
   appear in "PCI Devices" list in UI as devices with "QAT VF" in name.
   You can additionally verify same by running the `lspci` command in ESXi
   shell. For example:
    > lspci -vn | grep -E '4941|4943.*_VF_'
    0000:6b:00.1 Class 0b40: 8086:4941 [PF_0.107.0_VF_0]
    ....
    0000:e9:00.1 Class 0b40: 8086:4943 [PF_0.233.0_VF_0]
    ....

   Note: Actual output depends on the hardware available on the particular
   system.

   At this point the Intel® QAT VFs can be attached to a guest Virtual Machine
   (see section "Pass-through the PCI Device" in this document).


Pass-through the PCI Device
---------------------------
1) Login to the target ESXi host via vSphere Host Client.
2) In the left pane, click on VMs.
3) In the center pane, click on the desired Virtual Machine.
   Ensure that the VM is powered off.
4) Click on the "Edit" button to edit the virtual machines settings. A pop-up
   window with the VM settings will appear.
5) Click on "Add other Device" and select "PCI device". The new PCI device will
   be added. By default, it selects the first VF in the system, but it may
   not be a Intel® QAT VF since there could be other PCI devices present in the
   system. To select Intel® QAT VF, click the drop-down list and choose desired
   PCI device with "QAT VF" in name. The BDFs listed here will match with the
   output of the `lspci -vn | grep -E '4941|4943.*_VF_'` command. Additional
   VFs can be added by repeating this step.
6) All memory for the VM must be reserved. Expand "Memory" and set
   "Reserve all guest memory (All locked)" checkbox.
7) Click "Save".

Now you have one or more VFs attached to your guest and VM is ready to be
powered on. Refer to "Installing Intel® QAT Software on the Guest" section
for details about Guest driver.


Installing Intel® QAT Software on the Guest
==============================================================
For instructions on how to install the Guest driver on a Linux guest please
refer to "Using Intel® Virtualization Technology (Intel® VT) with
Intel® QuickAssist Technology Application Note" and corresponding
guest driver's collaterals.


Device configuration
====================
Intel® QAT HW support few different services. By default, each PF is configured
to provide specific set of services, but this can be changed via driver
module option `srv_mask`.
Next table represents what services are available for given HW, it's default
configuration and how many could be configured simultaneously:

  |  Device  | Available services | Default config | # of services |
  |----------|--------------------|----------------|---------------|
  | 4XXX QAT |   ASYM, SYM, DC    |   ASYM + DC    |       2       |

Bit positions of `srv_mask` parameter help in enabling a particular service:

  | Bit position |  7 to 4  |  3   | 2  |  1  |    0     |
  |--------------|----------|------|----|-----|----------|
  | Service      | Reserved | ASYM | DC | SYM | Reserved |

For example, to enable ASYM and SYM services, decimal value `10` == (2|8)
should be passed. Each device may have a specific configuration (order of
Intel® QAT devices is matching order of appearance in `lspci` output) -
to enable SYM configuration on first device, ASYM on second and default
configuration on other devices, next `srv_mask` should be passed - `2,8`.

To set device configuration as in example above, you need to execute
next command in ESXi Shell:
  > esxcli system module parameters set -m qat -p "srv_mask=2,8"

To view module parameters with descriptions and previously set values,
you need to execute next command in ESXi Shell:
  > esxcli system module parameters list -m qat

To reset module configuration to a default, you need to execute
next command in ESXi Shell:
  > esxcli system module parameters clear -m qat

Configuration could be applied only on module initialization, so after updating
device configuration you need to restart system or reload driver as described
in "Recover from fatal errors" section of this document to apply desired
configuration.

Attempt to use incorrect/unsupported configuration for device may prevent
driver from proper initialization and HW may become unavailable until
proper configuration will be used or be reset to default.

Due to fact that with virtualization we always have 2 drivers - Host and Guest,
both should be properly configured. Guest's driver configuration should
match the capabilities provided by Host driver.
More details about Guest driver configuration could be taken from it's
documentation like Release Notes or Getting Started Guide.


Uninstalling Intel® QAT VMware Driver
==================================
1) Open an SSH connection to the target ESXi host.
2) Actual process depends on the way how the driver was installed on system:
    a) If it was installed as component (recommended) - run next command
    to remove driver component:
        > esxcli software component remove -n Intel-qat
    b) If it was installed as VIB package - run next command to remove VIB:
        > esxcli software vib remove -n qat
3) Reboot system to complete removal:
    > reboot


Known issues
============
1) By default, VMkernel supports only 1024 interrupt cookies. On system with
   big number of QAT devices and other accelerators, interrupt cookies could be
   exhausted which may lead to various issues and QAT HW will be not available.
   Increase interrupt cookies number to the desired value (up to 4096) to
   support more QAT devices via next command:
       > esxcli system settings kernel set -s=maxIntrCookies -v=4096


Recover from fatal errors
=========================
In the event of persistent device error state that cannot be recovered
by software, it is recommended to manually reload the PF driver
on the ESXi host or reset the host itself.
The driver will reset and recover the PF device during driver reloading.
Steps to reload the driver:
1) Power off all the VMs that are using Intel® QAT hardware.
2) Execute next commands via SSH on ESXi host to reload the PF driver:
    > esxcfg-module -u qat
   and signal to ESXi device manager to rediscover HW:
    > kill -HUP $(cat /var/run/vmware/vmkdevmgr.pid)

Note: If driver is used together with Intel® Accelerator Management Daemon for
VMware ESXi*, driver unload may report that "module symbols in use". In this
case please proceed with system reboot or refer to daemon documentation
on how to temporally stop it.
